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PREFACE 


This  paper  documents  a  NATO  Aircrew  Selection  Working  Group 
(ACSWG)  project  to  (a)  identify  critical  ability  requirements  for  successful 
performance  in  fighter  aircraft  and  (b)  to  recommend  a  selection  test  battery 
to  assess  these  critical  abilities. 

These  authors  thank  the  fighter  pilots  who  served  as  subject  matter 
experts  in  the  ability  analysis  study.  Appreciation  also  is  extended  to  Maj 
Per  Byrdorf,  Dr  Ferdinand  Rameckers  and  Dr  Gail  Walker-Smith  who  served 
on  the  selection  instrument  review  board. 


THE  IDENTIFICATION  OF  ABILITY  REQUIREMENTS 
AND  SELECTION  INSTRUMENTS  FOR 
FIGHTER  PILOT  TRAINING 

SUMMARY 

Forty-three  experienced  fighter  pilots  from  Canada,  Norway,  and  the 
United  States  served  as  subject  matter  experts  (SMEs)  in  an  effort  to 
determine  the  relative  importance  of  27  personnel  characteristics  for  fighter 
pilot  performance.  Inter-rater  reliability  estimates  indicated  an  acceptable 
level  of  agreement  for  SMEs  within  each  country  and  between  pairs  of 
countries  regarding  the  relative  importance  of  the  27  characteristics.  Because 
there  was  sufficient  agreement  among  SMEs  from  the  three  countries,  the 
average  ranking  of  the  27  characteristics  was  calculated.  Based  on  these 
results,  aviation  psychologists  from  Canada,  Denmark,  the  Netherlands, 
Norway,  the  United  Kingdom,  and  the  United  States  reviewed  selection 
instruments  currently  in  use  in  NATO  member  countries,  to  identify  the  most 
promising  selection  instruments  for  inclusion  in  a  computer-based  fighter  pilot 
test  battery.  Their  recommendations  are  summarized  in  the  paper. 


INTRODUCTION 


Background 

The  early  efforts  of  the  Euro-NATO  Aircrew  Selection  Working  Group 
(ACSWG)  included  the  development  of  two  reference  manuals.  The  first, 
referred  to  as  the  Red  Book  (ACSWG,  1983),  provided  an  overview  of  member 
pilot  candidate  selection  and  screening  procedures  (i.e.,  computer-based  testing, 
personality  testing,  basic  predictors,  flight  screening).  The  second  manual, 
the  Black  Book  (ACSWG,  1988),  provided  detailed  information  regarding 
member  selection  instruments  (i.e.,  general  and  technical  characteristics, 
administration  procedures,  theoretical  basis,  technical  evaluation).  As  of  1982 
when  the  ACSWG  was  founded,  the  predominant  pilot  candidate  selection 
factors  within  NATO  included  medical  and  physical  fitness,  academic 
performance,  paper-and-pencil  aptitude  test  scores,  personality,  and  previous 
flying  experience.  Most  countries  reported  no  experience  with  computer-based 
tests  and  only  Germany  was  using  computer-based  tests  for  operational 
selection  in  1983  (ACSWG,  1983). 


Purpose 

One  of  the  early  goals  of  the  ACSWG  was  to  identify  ability  requirements 
for  Fighter  pilots.  This  was  accomplished  through  an  ability  analysis  of 
fighter  pilot  tasks  using  a  modified  version  of  the  Levine,  Mallamad,  and 
Fleishman  (1978)  decision-flow  diagram.  Following  the  completion  of  the 
ability  analysis,  ACSWG  aviation  psychologists  reviewed  pilot  candidate 
selection  instruments  currently  used  by  NATO  member  countries  to  recommend 
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a  selection  battery  for  fighter  pilots.  The  purpose  of  this  report  is  to 
document  the  results  of  the  ability  analysis  and  selection  instrument  review. 


STUDY  1:  ABILITY  ANALYSIS 


Method 


Subjects 

The  subject  matter  experts  (SMEs)  in  this  study  were  43  experienced 
fighter  pilots  from  Canada  (n  =  18),  Norway  (n  =  15),  and  the  United  States 
(n  =  10).  Their  expert  knowledge  of  the  task  requirements  were  crucial  for 
identifying  and  ranking  the  abilities  necessary  for  fighter  pilots. 


Procedure 

SMEs  performed  an  exercise  to  determine  the  importance  of  personnel 
characteristics  for  fighter  pilot  tasks  using  a  modified  version  of  the  Levine, 
Mallamad,  and  Fleishman  (1978)  decision-flow  diagram.  With  this  technique, 
SMEs  used  a  structured  series  of  questions  to  determine  the  importance  of 
characteristics  for  performing  12  job-related  tasks.  The  tasks  were  chosen 
to  be  unique  and  critical  to  flying  high-performance  fighter  aircraft.  Thus, 
pilot  tasks  that  are  common  to  other  types  of  aircraft  (e.g.,  receive  IFR 
clearance)  were  not  included.  Each  SME  rated  the  importance  of  27  abilities 
and  characteristics  for  performing  each  of  the  12  job-related  tasks.  Definitions 
of  the  27  abilities  and  characteristics  are  provided  in  Appendix  A  and  the 
12  pilot  tasks  are  summarized  in  Appendix  B. 

Each  SME  was  provided  with  a  package  containing  the  decision-flow 
diagram,  12  critical  pilot  tasks,  and  a  set  of  instructions.  Each  SME  was 
required  to  select  a  task,  and  while  considering  the  task,  work  through  the 
questions  in  the  decision-flow  diagram.  Whenever  the  SME  decided  a 
characteristic  was  necessary  for  successful  completion  of  the  task  under 
consideration,  its  importance  was  indicated  on  a  five-point  scale.  Once  all 
questions  in  the  decision-flow  diagram  had  been  completed  for  a  particular 
task,  a  new  task  was  selected  and  the  process  repeated.  Using  this  procedure, 
the  SMEs  independently  rated  the  importance  of  each  of  the  27  characteristics 
for  the  performance  of  each  of  the  12  critical  pilot  tasks.  For  additional 
details  regarding  the  decision-flow  diagram  and  questionnaire,  see  Rodgers 
and  Sage  (1986). 
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Results 


Inter-rater  reliability  estimates  (i.e.,  consistency  among  SMEs)  calculated 
separately  for  SMEs  from  each  participating  country,  indicated  an  acceptable 
level  of  agreement  for  SMEs  within  each  country  (Canada,  .91;  Norway,  .86; 
United  States,  .90).  SMEs  within  each  country  tended  to  rank-order  the  27 
abilities  in  a  similar  manner  in  terms  of  their  importance  for  performing  in 
a  fighter-type  aircraft. 

The  mean  score  for  each  characteristic  and  its  ranking  for  each  group  of 
SMEs  are  presented  in  Table  1.  Examination  of  the  mean  scores  for  each 
characteristic  across  countries  revealed  significant  differences  for  23  of  the 
27  characteristics.  SMEs  from  the  United  States  (M  -  3.65)  consistently 
rated  characteristics  to  be  of  greater  importance  than  did  SMEs  from  Canada 
(M  =  2.90)  or  Norway  (M  =  2.20).  It  was  not  determined  the  extent  to 
which  these  differences  were  due  to  response  bias  or  to  true  differences  in 
the  perceived  importance  of  these  characteristics.  The  source  of  the  mean 
differences  was  not  considered  important,  as  the  purpose  of  the  study  was 
to  determine  the  relative  importance  (i.e.,  ranking)  of  the  27  characteristics. 

Within  each  country,  characteristics  were  ranked  from  most  to  least 
important.  Rank-order  correlation  coefficients  (i.e.,  Spearman’s  rho)  calculated 
for  each  pair  cf  countries  indicated  substantial  agreement  regarding  the 
relative  importance  of  the  27  characteristics  (Canada-Norway,  .93; 
Canada-United  States,  .91;  Norway-United  States,  .93).  Because  there  was 
sufficient  agreement  among  SMEs  from  the  three  countries,  the  average 
rankings  of  the  27  characteristics  were  calculated  (see  Table  l). 


Discussion 

The  rating  of  the  characteristics  necessary  for  the  successful  performance 
of  fighter  pilot  tasks  was  shown  to  be  a  reliable  process  for  SMEs  from  each 
of  the  participating  countries.  Of  greater  importance  was  the  high  level  of 
agreement  regarding  the  relative  importance  of  these  characteristics  across 
the  three  groups  of  SMEs.  These  results  suggested  that  the  ranking  of  the 
importance  of  the  characteristics  was  coheremt  (i.e.,  the  most  important 
characteristics  can  be  clearly  identified)  and  has  important  implications  for 
the  development  of  pilot  candidate  selection  instruments.  These  results 
suggest  that  when  constructing  a  pilot  candidate  selection  battery,  emphasis 
should  be  placed  on  measures  of  situational  awareness  (i.e.,  readiness  to 
respond  to  situational  changes),  memorization  (i.e.,  memory  for  numbers, 
procedures,  etc.),  and  reasoning  (i.e.,  combining  information  to  form  logical 
conclusions).  Consideration  should  be  given  to  developing  measures  of  other 
characteristics  in  order  of  their  relative  importance  as  disclosed  by  the 
rankings. 
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TABLE  L  SUMMARY  OF  CHARACTERISTIC  RANKINGS  BY  COUNTRY 


Mean  and  Rank-Order 


Average  Canada  Norway  USA 


Ability 

Rank-Order 

Mean 

Rank 

Mean 

Rank 

Mean 

Rank 

Situational  Awareness 

1.67 

4.33 

1 

3.63 

2 

4  44 

2 

Memorization 

2.33 

4.04 

2 

3.73 

1 

4  42 

4 

Achievement  Motivation 

4.67 

3.65 

7 

2.89 

6 

4  51 

1 

Reasoning 

5.00 

3.71 

6 

3.21 

4 

4  41 

5 

Perceptual  Speed 

5.33 

3.72 

5 

3  17 

5 

4  37 

6 

Time  Sharing 

800 

3.44 

12 

3.52 

3 

4.43 

3 

Aggressiveness 

6.67 

3.88 

3 

2.71 

9 

4  20 

8 

Selective  Attention 

7.33 

3.64 

8 

2.84 

7 

4.29 

7 

Response  Orientation 

8.50 

3.75 

4 

2  64 

10 

4.13 

11 

Spatial  Orientation 

9.00 

3.53 

9 

2.77 

8 

4,14 

10 

Divided  Attention 

10.33 

3.48 

10 

2.26 

12 

4.19 

9 

Emotional  Stability 

13.33 

3.47 

11 

2.22 

13 

3  92 

16 

Psychomotor  Coordination 

13.33 

3.30 

15 

2.53 

11 

3  95 

14 

Flexibility  of  Closure 

14.17 

3.13 

16 

2.08 

15 

4  12 

12 

Information  Ordering 

14.33 

3.31 

14 

1.99 

16 

4  04 

13 

Risk  Taking 

14.67 

3.39 

13 

2.15 

14 

3  89 

17 

Cooperativeness 

16.67 

2.42 

18 

1.89 

17 

3  94 

15 

Control  Precision 

18.83 

2.53 

17 

1  66 

21 

3.59 

18 

Number  Facility 

20.33 

2.37 

19 

1.85 

18 

2  70 

24 

Assertiveness 

20.33 

2.22 

20 

1.63 

22 

3  58 

19 

Visualization 

20.50 

2.21 

21 

1.78 

19 

3.21 

22 

Oral  Comprehension 

22.17 

1.68 

23 

1.42 

23 

3  37 

20 

Rate  Control 

22.67 

2.16 

22 

1.71 

20 

1.54 

26 

Oral  Expression 

23.17 

1.67 

24 

1.14 

25 

3.35 

21 

Written  Comprehension 

24.00 

1.43 

25 

1.19 

24 

3.00 

23 

Leadership 

25.67 

1.33 

26 

.69 

25 

2.52 

25 

Written  Expression 

27.00 

53 

27 

14 

27 

1.23 

27 

These  results  were  consistent  with  those  from  similar  ability  analyses 
conducted  with  fighter  pilots  from  other  NATO  countries  (ACSWG,  1987)  and 
US  Army  helicopter  pilots  (McAnulty,  1987).  In  the  ACSWG  (1987)  study, 
SMEs  from  Denmark,  Germany,  the  Netherlands,  and  the  United  Kingdom 
rank-ordered  a  similar  list  of  characteristics  for  the  same  12  fighter-type 
tasks.  The  most  important  characteristics  in  descending  order  were  situational 
awareness,  spatial  orientation,  time  sharing,  aggressiveness,  divided  attention, 
psychomotor  coordination,  perceptual  speed,  selective  attention,  and 
visualization.  The  most  notable  difference  between  these  results  and  those 
reported  in  Table  1,  was  that  “memorization”  was  not  among  the  most 
important  characteristics  identified  in  the  ACSWG  (1987)  study.  This  may 
have  been  because  a  different  definition  of  memorization  was  used  in  these 
two  studies.  In  the  ACSWG  (1987)  study,  “memorization”  emphasized  long-term 
memory  whereas  the  focus  in  this  study  was  on  short-term  memory  processes. 
Similar  definitional  differences  occurred  for  the  attributes  “achievement 
motivation”  and  “assertiveness.” 
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Despite  marked  differences  in  the  performance  characteristics  of  fixed-wing 
and  rotary  wing  aircraft,  McAnulty  (1987)  identified  a  similar  set  of  crucial 
characteristics  for  helicopter  pilots.  The  most  important  characteristics  for 
primary  and  instrument  helicopter  training  included  perceptual  speed,  control 
precision,  multilimb  coordination,  time  sharing,  arm/hand  steadiness,  oral 
comprehension,  deductive  reasoning,  speed  of  closure,  selective  attention, 
choice  reaction  time,  and  spatial  orientation. 

The  term  “situational  awareness”  was  not  used  explicitly  in  the  McAnulty 
(1987)  study.  However,  its  components  were  included  in  the  combination  of 
“speed  of  closure”  and  “choice  reaction  time”  abilities.  In  a  similar  manner, 
the  characteristic  “psychomotor  coordination”  was  covered  by  the  more  specific 
characteristics  of  “multilimb  coordination,”  and  ‘arm/hand  coordination.” 


Conclusion 

Results  from  the  current  a  .id  previous  studies  suggest  that  a  set  of 
common  characteristics  are  necessary  for  fixed  wing  and  rotary  wing  pilots. 
Although  some  differences  were  noted  between  average  rated  fighter  and 
helicopter  pilot  characteristics,  selection  test  batteries  for  these  two  groups 
should  have  considerable  overlap. 

Based  on  these  results,  a  pilot  candidate  selection  test  battery  should 
include,  at  a  minimum,  measures  of  situational  awareness,  memorization, 
and  reasoning.  Measures  of  other  characteristics  should  be  targeted  for 
inclusion  to  the  extent  to  which  they  are  expected  to  provide  incremental 
validity  (i.e.,  account  for  unique  prediction  of  the  criterion). 


STUDY  2:  SELECTION  INSTRUMENT  REVIEW 


Method 


Subjects 

Aviation  psychologists  from  Canada,  Denmark,  the  Netherlands,  Norway, 
the  United  Kingdom,  and  the  United  States  served  as  subject  matter  experts 
rSMEs).  Their  expert  knowledge  of  the  targeted  pilot  characteristics,  testing 
issues  (i.e.,  reliability,  validity,  interpretability),  and  the  candidate  selection 
instruments  were  crucial  for  identifying  the  most  promising  selection 
instruments  for  inclusion  in  a  computer-based  fighter  pilot  test  battery. 
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Procedure 


Study  1  focused  on  determining  important  fighter  pilot  characteristics  like 
cognitive  and  perceptual-motor  abilities  (e.g.,  memorization,  psychomotor 
coo:  ’ination)  and  personality  constructs  (e.g.,  achievement  motivation, 
risk-taking  ).  Study  2  focused  on  identifying  selection  instruments  that  could 
be  used  to  measure  individual  differences  in  the  12  most  important  abilities. 
The  identification  of  suitable  measures  of  important  personality  constructs 
was  set  aside  as  a  separate  task. 

Before  the  selection  instrument  review  could  begin,  it  was  necessary  to 
gather  information  about  the  test  characteristics  of  potential  selection 
instruments  (i.e.,  those  currently  in  use  in  the  national  pilot  selection 
programs).  Aviation  psychologists  from  each  participating  country  were 
requested  to  document  national  pilot  selection  instruments  using  a  structured 
format  which  included  information  regarding  general  characteristics  (e.g.,  title, 
publisher),  technical  characteristics  (e.g.,  number  of  items,  scoring  procedure), 
administration  of  the  test,  psychological  data  (e.g.,  theoretical  basis),  technical 
evaluation  (e.g.,  norms,  reliability,  validity),  references,  and  a  summary 
evaluation.  For  additional  details  see  Appendix  C. 

The  test  review  board,  which  consisted  of  aviation  psychologists  from 
Canada,  Denmark,  the  Netherlands,  Norway,  the  United  Kingdom,  and  the 
United  States,  considered  several  factors  when  reviewing  the  candidate  selection 
instruments.  Tests  were  judged  for  ease  of  programmability,  dimensionality 
(i.e.,  does  test  measure  one  or  more  than  one  attribute), 
scoreability/interpretability,  validity,  culture  fairness,  and  other  factors.  Table 
2  provides  a  complete  list  of  the  factors  considered.  Some  of  the  factors 
listed  in  Table  2  could  cause  a  test  to  be  rejected  without  consideration  of 
other  factors  v^.g.,  lack  of  copyright  ownership,  lack  of  culture  fairness, 
non-programmable).  Tests  that  were  judged  to  meet  the  minimum 
requirements  for  inclusion  were  compared  in  greater  detail.  For  example, 
short  tests  generally  were  considered  more  desirable  than  long  tests  with 
similar  programming  requirements  and  validity. 


TABLE  2.  FACTORS  USED  TO  ASSESS  TEST  SUITABILITY  FOR 
INCLUSION  IN  TITE  ACSWG  TEST  BATTERY 


1.  Programmability 

2.  Dimensionality 

3.  Advantage  of  computer  over  paper-and-poncil  administration 

4.  Sco  eability/Interpretability 

5.  Data  reduction  complexity 

6.  Test  length 

7.  Construct  and  predictive  validity 

8.  Culture  fairness 

9.  Number  of  countries  where  test  is  currently  in  use 

10.  Copyright  ownership 

11.  Machine  or  system  dependency 

12.  Aptitude  (rather  than  achievement)  test 

13.  Uniqueness  of  test  (i.e.,  is  test  the  same  as  others  being  considered) 
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Results 


Ninety  Seven  (97)  tests  from  nine  countries  were  evaluated.  A  summary 
of  the  evaluations  including  the  test  name,  abilities  measured,  country 
identification  code,  and  suitability  decision  are  summarized  in  Appendix  D. 
When  a  test  was  judged  to  be  unsuitable  for  further  consideration,  the 
factor(s)  related  to  this  decision  were  noted.  For  example,  BE2  (Reasoning 
Test)  was  judged  to  lack  culture  fairness,  NO  13  (Mirror  Tracing)  was  not 
programmable,  and  US6  (Time  Sharing)  was  too  long. 

The  21  tests  judged  to  be  suitable  for  inclusion  in  the  computer-based 
test  battery  are  listed  in  Appendix  E.  The  targeted  test  battery  length  was 
two  hours.  As  it  was  neither  feasible  nor  desirable  to  include  all  of  these 
in  the  test  battery,  they  were  further  scrutinized  to  identify  those  with  the 
greatest  potential.  The  21  tests  were  evaluated  in  terms  of  the  abilities 
measured,  ease  of  programmability,  predictive  validity,  and  test  length.  The 
final  recommended  tests  are  summarized  in  Table  3.  Individual  test 
descriptions  are  presented  in  Appendix  F. 


TABLE  3.  AIRCREW  SELECTION  WORKING  GROUP  FIGHTER 
PILOT  TEST  BATTERY  SUMMARY 


Length 

Test  Name  (mins)  Attribute  Measured  Types  of  Scores 


Test  Battery  Introduction 

10 

Biographical  information 

Age,  gender, 
previous  flying 
experience 

Vigilance 

10 

Situational  awareness,  time¬ 
sharing,  divided  attention 

Number  of  routine 
tasks,  number  of 
priority  tasks, 
response  time  on 
priority  tasks 

Matrices 

15 

Reasoning  (nonverbal) 

Response  time, 
response  accuracy 

Digit  Recall 

5 

Memorization 

Response  time, 
response  accuracy, 
weighted  accuracy 
score 

Complex  Coordination 

10 

Time-sharing,  psychomotor 
coordination  (compensatory 
tracking) 

Tracking  error 

Instrument  Comprehension 

20 

Reasoning,  visualization 

Response  time, 
response  accuracy 

Time-Sharing  2 

15 

Time-sharing,  divided 
attention,  control  precision 

Tracking  difficulty’, 
response  time, 
response  accuracy 

Scheduling  2 

6 

Situational  awareness,  time 
sharing,  divided  attention 

Number  of  points 
accumulated 

DTG 

20 

Selective  attention,  response 
orientation 

Response  accuracy 

No  tests  of  selective  attention  or  response  orientation  suitable  for 
computer-based  administration  without  customized  keypads  or  peripheral 
devices  were  identified.  The  SMEs  recommended  that  the  ACSWG  consider 
leasing  or  acquiring  specialized  equipment  to  host  tests  measuring  selective 
attention  and  response  orientation  (e.g.,  Determinations  Geraet  or  DTG  test 
device). 


CONCLUSION 

The  ACSWG  adopted  the  recommendations  of  the  SMEs  for  the  development 
of  a  computer-based  fighter  pilot  selection  battery.  The  ACSWG  subsequently 
forwarded  this  information  to  the  Air  Force  Subgroup  (AFSG)  which  oversees 
ENJJPT.  The  ACSWG  then  requested  and  was  granted  AFSG  permission  to 
validate  the  recommended  test  battery  against  ENJJPT  performance  criteria. 

The  ACSWG  fighter  pilot  selection  battery  was  programmed  on  modified 
Portable  Basic  Attributes  Test  systems  (i.e.,  PortaBAT;  see  Carretta,  1987  for 
a  description).  Rudder  pedals  on  loan  from  the  UK  were  added  for  use  in 
the  Complex  Coordination  test.  As  the  modified  PortaBAT  systems  were  not 
suitable  for  administering  tests  of  selective  attention  or  response  orientation, 
the  ACSWG  acquired  two  DTG  systems  (on  loan  from  Dr  Schufried)  to 
measure  these  abilities. 

Six  PortaBAT  systems  are  on  loan  from  the  US  Air  Force  to  the  ACSWG 
for  use  during  the  fighter  pilot  test  battery  validation.  Test  instructions 
have  been  programmed  in  English,  Italian,  Portuguese,  Spanish,  and  Turkish 
to  accommodate  national  testing  requirements.  Test  battery  validation  is 
expected  to  be  completed  in  Fall  1994. 
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APPENDIX  A 


DEFINITIONS  OF  CHARACTERISTICS  USED  IN  FIGHTER  PILOT  TASK  ANALYSIS 

ORAL  COMPREHENSION:  The  ability  to  understand  spoken  English  words  and  sentences. 

WRITTEN  COMPREHENSION:  The  ability  to  understand  written  sentences  and  paragraphs. 

ORAL  EXPRESSION:  The  ability  to  use  English  words  or  sentences  in  speaking  so  others 

will  understand. 

WRITTEN  EXPRESSION:  The  ability  to  use  English  words  or  sentences  in  writing  so  others 

will  understand. 

MEMORIZATION:  The  ability  to  remember  information,  such  as  words,  numbers,  pictures, 
and  procedures.  Bits  of  information  can  be  remembered  by  themselves 
or  with  other  pieces  of  information. 

SITUATIONAL  AWARENESS:  The  state  of  constant  mental  readiness  in  order  to  respond  to 

situational  changes. 

REASONING:  The  ability  to  combine  separate  bits  of  information  and  to  apply  general  rules 
in  order  to  derive  logical  answers  or  form  conclusions. 

INFORMATION  ORDERING:  The  ability  to  correctly  follow  a  rule  or  set  of  rules  to  arrange 

things  or  actions  in  a  certain  order.  The  rule  or  set  of  rules 
used  must  be  given.  The  things  or  actions  to  be  put  in  order 
can  include  numbers,  letters,  words,  pictures,  sentences,  and 
mathematical  or  logical  operations. 

NUMBER  FACILITY:  Involves  the  degree  to  which  adding,  subtracting,  multiplying,  and 

dividing  can  be  done  quickly  and  correctly.  These  can  be  steps  in 
other  operations  like  finding  percentages. 

DIVIDED  ATTENTION:  The  ability  to  shift  back  and  forth  between  two  or  more  sources  of 

information. 

FLEXIBILITY  OF  CLOSURE:  The  ability  to  identify  relevant  information  in  a  complex 

perceptual  field. 

TIME  SHARING:  The  ability  to  observe  several  sources  of  information,  actions  or  tasks  at 
the  same  time,  to  combine  them,  allot  task  priorities,  and  integrate  them 
into  actions  that  have  to  be  performed. 

SELECTIVE  ATTENTION:  The  ability  to  concentrate  on  the  task  one  is  doing. 
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PERCEPTUAL  SPEED:  The  ability  to  perceive  quickly  and  accurately  even  small  details 

in  patterns  and  configurations. 

SPATIAL  ORIENTATION:  The  ability  to  tell  where  you  are  in  relation  to  the  location  of 

some  object  or  tell  where  the  object  is  in  relation  to  you. 

VISUALIZATION:  The  ability  to  imagine  the  movement  of  objects  in  three  dimensional 
space. 

PSYCHOMOTOR  COORDINATION:  The  ability  to  coordinate  movements  of  two  or  more 

limbs,  such  as  in  moving  equipment  controls.  Two  or 
more  limbs  are  in  motion  while  the  individual  is 
sitting,  standing,  or  lying  down. 

CONTROL  PRECISION:  The  ability  to  move  controls  of  a  machine  or  vehicle.  This 

involves  the  degree  to  which  these  controls  can  be  moved  quickly 
and  repeatedly  to  exact  positions. 

RATE  CONTROL:  The  ability  to  adjust  an  equipment  control  in  response  to  changes  in  the 
speed  and/or  directions  of  a  continuously  moving  object  or  scene.  The 
ability  does  not  extend  to  situations  in  which  both  the  speed  and  direction 
of  the  object  are  perfectly  predictable. 


RESPONSE  ORIENTATION:  The  ability  to  choose  between  two  or  more  movements 

quickly  and  accurately  when  two  or  more  different  signals 
(light,  sounds,  pictures)  are  given.  The  ability  is  concerned 
with  the  speed  with  which  the  right  response  can  be  started 
with  the  hand,  foot,  or  other  parts  of  the  body. 


AGGRESSIVENESS:  The  ability  to  decide  rapidly  on  an  adequate  appropriate  action  and  to 

carry  it  out  immediately. 


RISK  TAKING:  Willingness  to  take  bold  decisions  based  on  an  adequate  cognition  of 
dangerous  situations.  Not  possessing  a  defensively  cautious  attitude. 

ASSERTIVENESS:  Belief  in  one’s  own  capabilities.  Self-assured  and  possessing  a 

willingness  to  defend  one’s  own  opinions. 

EMOTIONAL  STABILITY:  Emotionally  mature,  stable,  and  having  few  neurotic  symptoms. 
COOPERATIVENESS:  Willingness  to  cooperate  with  other  people. 
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LEADERSHIP:  The  ability  to  lead  others  based  on  personal  behavior,  authority,  and  the 
ability  to  convince  others.  Personnel  management  skills. 

ACHIEVEMENT  MOTIVATION:  Willingness  and  determination  to  work  towards  goals. 
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APPENDIX  B 


CRITICAL  FIGHTER  PILOT  TASKS 

1.  Perform  systems/weapons  checks. 

2.  Manage  on-board  systems. 

3.  Set  up  attack. 

4.  Perform  tactical  offensive  flight  maneuvers. 

5.  Avoid,  evade  or  suppress  threats. 

6.  Monitor  and  control  flight  parameters. 

7.  Perform  weapons  delivery. 

8.  Perform  formation  tactics. 

9.  Respond  to  aircraft  emergency  situations. 

10.  Manage  communications. 

11.  Perform  low  level  navigation. 

12.  Perform  tactical  defensive  flight  maneuvers. 
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APPENDIX  C 

STANDARDIZED  TEST  DESCRIPTION  FORM 
1.0  General  Characteristics 

1.1  Title  of  test 

1.2  Author 

1.3  Publisher 

1.4  Complementary  materials 

1.5  Country  of  origin 

1.6  Language  of  origin 

1.7  Other  language  versions 

2.0  Technical  Characteristics 

2. 1  Number  of  items 

2.1.1  Item  type 

2.2  Administration  mode 

2.3  Time  required  to  administer  test 

2.4  Scoring  procedure 

2.5  Time  required  to  score  or  interpret  test 

2.6  Examiner  qualifications 

2.7  Type  of  test 

2.8  Subtests 

2.9  Parallel  Forms 

2.10  Age  groups 

3.0  Admninistration  of  the  test 

3.1  Facilities 

3.2  Material 

3.3  Directions 
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4.0  Psychological  Data 


4.1  Theoretical  basis 

4.2  Nature  of  content 

5.0  Technical  Evaluation 

5.1  Norms 

5.1.1  Type 

5.1.2  Standardization  Sample 

5.2  Reliability 

5.2.1  Types  of  procedures 

5.2.2  Scorer  reliability 

5.2.3  Equivalence  of  forms 

5.2.4  Long-term  stability 

5.3  Validity 

5.3.1  Types  of  validation  procedures  used 

5.3.2  Specific  procedures  followed  in  assessing  validity  and  results  obtained 

5.3.3  Size  and  nature  of  validation  sample 


6.0  References  -  list  of  studies  related  to  test 

7.0  Summary  of  Evaluation-Strengths  and  weaknesses  of  the  test 
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APPENDIX  D 


TESTS  CONSIDERED  FOR  INCLUSION  IN  ACSWG 
FIGHTER  PILOT  TEST  BATTERY 


Test  Name 

Instrument  Comprehension 
Reasoning  Rennes 
Mechanical  Comprehension 
Map  Reading  BAF304 
Coordination  Test 
Reaction  Meter  Type  318 
PICAR 

Determination  Gerat  DTG 
Two  Hand  Coordination 
Complex  Coordination 
Mental  Rotation 
Item  Recognition 
Imm&'late/Delayed  Memory 
Time  Sharing 
Vigilance 

Scanning  and  Allocation 
Manikin 

Pattern  Recognition 
Scheduling 

Serial  Mental  Arithmetic 
Kinesthetic  Memory 
Anticipation 
Persistance 

Instrument  Comprehension 
Time  Sharing  II 
Perceptual  Speed  II 
3D  Mental  Rotation 
ABCD  Working  Memory 
Inst  Coord  Analyser 
Sensory  Motor  Test 


Country 


_ Abilities  Measured1 

1  2  3  4  5  6  7  8  9  10  11  12 


Decision^ 
&  Reason 


«  *  •  *  *  * 


*  *  *  *  * 


*  * 


N-1,11 


N-U0.11 


N-1,6,11 
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Abilities  Measured1  Decision2 


Test  Name 

Country 

I  2 

3 

4 

5  6  7  8 

9 

10 

11  12 

&  Reason 

LGT3 

GE3 

* 

N-6.10 

KBT 

GE4 

* 

N-l 

TVT  Technical  Comprehension 

GE5 

• 

N-12 

Digit  Search  Test 

GE6 

* 

Y 

Labyrinth 

GE7 

• 

N-l 

Road  Figure  Test 

GE8 

* 

N-7 

Culture  Fair  IQ  Test 

GE9 

* 

N-l  0.8 

ABC-1  Logical  Reasoning 

UKl 

* 

N-8 

ABC-2  Logical  Reasoning 

UK2 

• 

* 

* 

N-8 

Angles  Bearings  Degrees 

UK3 

• 

N-7 

Visual  Search 

UK4 

* 

Y 

Dual  Task 

UK5 

* 

* 

N-13(US17) 

Manikin 

UK6 

* 

N-13(US9) 

Vigilance 

UK7 

* 

• 

N-13(US7) 

Digit  Recall 

UK8 

* 

Y 

Digit  Recognition 

UK9 

• 

N-130JK7) 

Annett  Handedness 

UK  10 

• 

N-7 

Time  Sharing 

UKll 

+  * 

* 

* 

Y 

Visualization 

UK  12 

• 

N-13(US19) 

Cubes 

UK  13 

* 

N-7 

Ravens  Adv  Prog  Matrices 

UK  14 

• 

N-10 

Sensory  Motor  Apparatus 

UK  15 

* 

N-13(BE5) 

Control  of  Velocity 

UK16 

* 

Y 

Instrument  Comprehension 

UK17 

* 

* 

• 

N-13(BE1) 

Mathematics  MATAB2 

UK  18 

* 

N-12 

TSM  Matrices 

NOl 

* 

Y 

Number  Series 

N02 

* 

N-6 

Mechanical  Comprehension 

N03 

* 

* 

N-12 

Pattern  Comprehension 

N04 

• 

N-10 

Figure  Pattern 

N05 

* 

Y 

Figure  Form 

N06 

* 

* 

N-13(US3) 

Time  Estimation  &  Blocks 

N07 

* 

« 

N-7 

Sorting  Test 

N08 

* 

* 

N-l 
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Test  Name 

Country 

Abilities  Measured1 

1  234567  89  10  11 

12 

Decision2 
&  Reason 

Direction  Reversal 

N09 

*  * 

N-l 

Number  Counting 

NO  10 

* 

N-13IGE6) 

Instrument  Comprehension 

NOll 

* 

• 

• 

N-13(BE1) 

Space 

N012 

* 

* 

+ 

N-7 

Mirror  Tracing 

NOD 

* 

* 

N-l 

Memory  Test 

NOW 

• 

N-7 

Instrument  Reading  WC2 

CA1 

* 

* 

* 

N-13(BE1) 

Serial  Addition  AS2 

CA2 

* 

N-12 

Vis  Gen  Aviat  VGAT 

CA3 

* 

N-ll 

General  Classification 

CA4 

* 

N-8 

Table  Reading  WT2 

CA5 

* 

N-l 

Instrument  Reading  WI2 

CA6 

* 

N-7 

Verbal  Aptitude  TV2 

CA7 

* 

N-8 

Maths  Reasoning  AR2 

CA8 

* 

N-7,12 

Auto  Pilot  Sel  Sym  CAPSS 

CA9 

• 

N-ll 

D48  Matrices 

IT1 

• 

N-10 

Deux  Barrage 

IT2 

*  * 

N-10 

GATB-3  Object  Comparison 

NE1 

* 

N-10 

GATB-5  Form  Comparison 

NE2 

* 

N-10 

GATB-7  Perceptual  Speed 

NE3 

• 

N-10 

Berenachot  G  Test 

NE4 

« 

N-8,10 

VMA  Aircraft  Manoeuvres 

NE5 

* 

Y 

Sensory  Motor  Apparatus 

NE6 

* 

N-13 

RBT  Radar  Display  Test 

NE7 

• 

N-7 

Instrument  Interpretation 

NE8 

• 

N-13(BE1) 

SVK  Heading  Estimation 

NE9 

* 

N-7 

Rudder  Control  Test 

NE10 

* 

N-ll 

Complex  Coordination 

NE11 

* 

N-l  1,1 

DTG-3 

NE12 

• 

N-l  1,10,1 

Raven  Matrices 

DK1 

* 

N-10 

IDP-53 

DK2 

* 

* 

N-8 

Mechanical  Comprehension 

DK3 

* 

N-10 

PIT 

DK4 

• 

N-13(NL5) 
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_ Abilities  Measured1  Decision2 

Test  Name  Country  123456789  10  11  12  &  Reason 


SIMCAP  DK3  •  N-l 


Notes 


Ability  Names 

1.  Situational  Awareness 

2.  Memorization 

3.  Reasoning 

4.  Perceptual  Speed 

5.  Time  Sharing 

6.  Selective  Attention 


7.  Response  Orientation 

8.  Spatial  Orientation 

9.  Divided  Attention 

10.  Psychomotor  Coordination 

1 1.  Control  Precision 

12.  Visualization 


2See  Table  2  for  an  explanation  of  'reason  for  non-suitability.* 
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APPENDIX  E 


TESTS  SUITABLE  FOR  INCLUSION  IN  ACSWG 
FIGHTER  PILOT  TEST  BATTERY 


DK  GE  BE  UK  NO  NL  CA  US  Test  Name  I  2  3  4  5  6  7  8  9  10  11  12 


17 

11 

8 

1  6 

Instrument  Comprehension 

•  * 

•  * 

15 

6 

Sensory  Motor  Apparatus 

* 

1 

Two  Hand  Coordination 

* 

2 

Complex  Coordination 

* 

• 

6 

3 

Mental  Rotation 

* 

4 

Item  Recognition 

• 

7 

7 

Vigilance 

* 

8 

Scanning  k  Allocation 

•  • 

*  * 

6 

9 

Manikin 

• 

Sensory  Motor  Test 

*  * 

10 

Digit  Search  Test 

* 

5 

17 

Time  Sharing  2 

• 

•  • 

18 

Perceptual  Speed  2 

• 

12 

19 

Mental  Rotation 

♦ 

4 

Visual  Search 

* 

8 

Digit  Recall 

• 

11 

Scheduling 

•  *  * 

* 

16 

Control  of  Velocity 

* 
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APPENDIX  F 


ACSWG  TEST  BATTERY  DESCRIPTIONS 
Test  Battery  Introduction 

This  interactive  subprogram  prompts  the  subject  to  provide  background  information  (e.g., 
identity,  age,  gender)  as  well  as  previous  flying  experience. 

Vigilance 

A  9  block  grid  appears  on  the  screen.  The  numbers  along  the  left  side  of  the  grid  are  the  "A" 
coordinates  and  correspond  to  the  rows.  The  numbers  along  the  top  of  the  grid  are  "B" 
coordinates  and  correspond  to  the  columns.  Each  block  in  the  grid  can  be  identified  by  its 
row  and  column  coordinates. 

During  the  test,  asterisks,  appear  within  the  blocks  of  the  grid.  The  subject’s  "routine 
task"  is  to  cancel  these  asterisks  as  quickly  as  possible.  An  asterisk  is  canceled  (i.e.,  erased) 
by  entering  its  row  and  column  coordinates  on  the  keypad. 

In  addition  to  the  asterisks,  arrows  "A"  may  appear  in  the  blocks  of  the  grid.  These 
arrows  represent  an  "emergency  task."  The  subject  is  instructed  to  respond  as  quickly  as 
possible  when  an  arrow  (i.e.,  emergency  )  appears.  This  function  is  done  by  pressing  the 
ENABLE  key,  then  entering  the  row  and  column  coordinates  of  the  arrow  (i.e.,  emergency). 
The  subject  is  instructed  to  resume  performing  the  routine  task  (i.e.,  canceling  asterisks), 
when  no  arrows  are  present 

The  measure  of  interest  for  this  test  are  the  number  of  routine  and  emergency  tasks 
successfully  completed  and  the  average  response  time  required  to  complete  emergency  tasks. 

The  psychological  factors  assessed  by  this  test  include  situational  awareness,  time  sharing, 
and  divided  attention.  The  test  requires  about  10  min  to  complete. 

Matrices 

In  the  Matrices  test,  a  picture  of  an  incomplete  geometric  pattern  appears  on  the  screen. 
(The  lower  right-hand  comer  of  the  pattern  is  missing).  The  subject’s  task  is  to  choose  from 
several  alternatives,  which  would  correctly  complete  the  pattern.  The  subject  indicates  his/her 
choice  by  entering  the  number  of  the  chosen  alternative  on  the  response  keypad. 

The  first  6  items  are  for  the  practice  only.  The  remaining  30  items  are  test  items  and  arc 
scored.  In  this  test  and  in  the  Instrument  Comprehension  test  (see  below),  the  subject  may  go 
backwards  and  forwards  through  the  test  The  subject  has  the  option  to  skip  items,  to  review 
the  items  previously  answered  or  skipped,  and  to  change  answers.  This  test  has  a  10-min 
time  limit  that  begins  when  the  subject  starts  the  first  test  item  (i.e.,  review  of  instructions 
and  practice  items  are  not  timed).  Response  speed  and  accuracy  are  recorded  on  each  item. 
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The  Matrices  test  assesses  non-verbal  reasoning  and  requires  about  15  min  to  complete. 

Digit  Recall 

In  this  test,  a  number  string  appears  on  the  screen.  After  a  few  seconds,  the  number 
string  is  removed  from  the  screen  and  is  replaced  by  a  string  of  empty  boxes.  (The  number 
of  boxes  is  equal  to  the  number  of  numbers  in  the  string).  The  subject’s  task  is  to  enter  the 
number  string  into  the  boxes. 

The  length  of  the  strings  vary  from  7  to  12  numbers.  Response  time  and  response 
accuracy  are  recorded  on  each  of  the  30  test  items.  Response  accuracy  is  calculated  in  3 
ways  for  each  item:  (1)  correct  or  incorrect,  (2)  number  of  numbers  placed  in  their  correct 
position  in  the  string,  and  (3)  a  weighted  scoring  algorithm  that  gives  partial  credit  for 
numbers  placed  out  of  sequence. 

This  test  measures  short-term  memory  and  requires  about  10  min  to  complete. 

Complex  Coordination 

The  ACSWG  Complex  Coordination  test  is  a  variation  of  the  USAF  test  with  the  same 
name.  In  this  test,  a  dual-axis  (right  hand)  control  stick  is  used  to  control  the  horizontal  and 
vertical  movement  of  a  cursor.  Rudder  pedals  (instead  of  the  left-hand  control  stick  in  the 
USAF  version)  are  used  to  control  the  left-right  movement  of  a  vertical  "rudder  bar"  of  light 
at  the  base  of  the  screen.  The  subject’s  task  is  to  maintain  the  cursor  (against  a  constant 
horizontal  and  vertical  rate  bias)  centered  on  a  large  cross  fixed  at  the  center  of  the  screen 
while  simultaneously  centering  the  rudder  bar  at  the  base  of  the  screen  (also  against  a 
constant  rate  bias).  After  receiving  instructions,  the  subject  completes  a  3-min  practice 
session  and  a  5-min  test.  The  Complex  Coordination  test  assesses  psychomotor  coordination 
and  time  sharing  ability  (i.e.,  compensatory  tracking  ability  involving  multiple  axes)  and 
requires  about  10  min  to  complete. 

Instrument  Comprehension 

In  this  test,  an  illustration  of  an  airplane  in  5  different  positions  is  shown  on  the  screen. 
An  artificial  horizon  indicator  and  a  compass  are  displayed  above  these  aircraft.  The 
subject’s  task  is  to  determine  which  of  the  aircraft  agrees  with  the  readings  on  the  artificial 
horizontal  indicator  and  compass.  The  subject  indicates  his/her  choice  on  each  item  by 
pressing  the  numbered  key  that  corresponds  to  the  chosen  alternative. 

The  test  items  begin  after  the  subject  has  completed  the  instructions  and  practice  items. 
As  with  the  Matrices  test  (see  above),  the  subject  may  go  backwards  and  forwards  through 
the  test,  and  may  skip  items,  review  items  previously  answered  or  skipped,  or  change 
answers. 

This  test  has  60  test  items  and  a  1 5-min  time  limit  that  begins  when  the  subject  starts  the 
first  test  item  (i.e.,  review  of  instructions  and  practice  items  are  not  timed). 
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Response  speed  and  response  accuracy  are  recorded  on  each  item.  The  instrument 
Comprehension  test  measures  reasoning  and  visualization.  Total  test  administration  time, 
including  instructions,  practice,  and  test  items  is  about  20  min. 

Time-Sharing  2 

Two  distinctly  different  kinds  of  tasks  are  involved  in  this  test.  The  first  is  a  measure  of 
hand-eye  coordination  and  the  second  is  a  measure  of  attention. 

The  first  three  1-min  trials  involve  tracking  only  to  provide  a  pure  estimate  of  the 
subject’s  psychomotor  coordination.  During  these  trials,  a  "stationary  image"  of  an  aircraft 
and  a  "gunsight"  that  move  to  the  left  or  right  are  displayed  on  the  screen.  The  subject  must 
maneuver  the  right-hand  control  stick  to  keep  the  gunsight  centered  on  the  airplane. 

The  next  two  1-min  trials  involve  detecting  and  responding  to  missing  numbers.  The 
numbers  appear  one  at  a  time  in  sequence  on  the  lower  part  of  the  screen.  The  number 
sequence  is  0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  0,  1,  2,  etc.  Occasionally  a  number  will  be  missing 
from  the  sequence  (e.g.,  0,  1,2,  3,  4,  6,  7,  ...[5  is  missing]).  The  subject’s  task  is  to  type  the 
missing  number  on  the  response  keypad.  Subjects  are  scored  on  both  response  speed  and 
accuracy. 

The  final  5  1-min  trials  involve  both  tracking  and  missing  digits.  While  the  subject  is 
maneuvering  the  right-hand  control  stick  to  keep  the  gunsight  on  the  airplane,  he/she  also 
must  monitor  the  number  counter  in  order  to  be  able  to  detect  the  missing  numbers. 

This  test  assesses  the  psychological  factors  of  time-sharing,  divided  attention,  and  control 
precision.  Test  administration  time  is  about  15  min. 

Scheduling  2 

In  this  test,  the  subject  is  presented  with  5  horizontal  scales  that  can  range  in  value 
between  0  and  10  points.  Each  scale  increases  at  a  unique,  constant  rate.  Each  scale  appears 
on  a  separate  screen  and  may  be  viewed  by  entering  the  scale  number  on  the  response 
keypad.  (1,  2,  3,  4,  or  5).  The  subject  "scores"  points  equal  to  the  current  value  of  the 
displayed  scale  by  pressing  the  ENABLE  key.  When  the  ENABLE  key  is  pressed,  the 
current  value  of  the  displayed  scale  is  added  to  the  subject’s  total  score  and  the  scale  is  reset 
to  0  (where  it  will  start  incrementing  again). 

If  the  value  of  a  scale  reaches  its  upper  limit,  and  the  subject  has  not  responded  by 
pressing  the  ENABLE  key,  the  value  of  the  scale  will  return  to  0  without  the  subject 
receiving  any  points  for  that  scale. 

The  Scheduling  2  test  measures  the  psychological  constructs  of  situational  awareness, 
time-sharing,  and  divided  attention.  Tins  test  requires  about  6  min  to  complete. 
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Determinants  Geraet 


This  test  is  hosted  on  the  DTG  test  device  rather  than  the  Porta-BAT  system.  The 
subject’s  task  is  to  respond  as  quickly  and  accurately  as  possible  to  auditory  (i.e.,  high  and 
low  tones)  and  visual  signals  (i.e.,  colored  lights)  by  pressing  foot  pedals  and/or  buttons. 
Subjects  are  presented  with  detailed  instructions  as  to  how  to  respond  to  different  visual  and 
auditory  signals. 

The  test  begins  with  several  practice  items  that  are  neither  timed  nor  scored.  The  purpose 
of  the  practice  items  is  for  the  subject  to  learn  the  appropriate  response  to  each  stimulus. 

After  reviewing  the  instructions  and  completing  the  practice  items,  the  subject  is  presented 
with  a  series  of  test  items.  The  test  consists  of  5  groups  of  items.  There  are  150  items  in 
Group  1  and  75  items  in  Group  2  through  5  (150  +  4  (75)  =  450  items).  As  described  earlier, 
during  the  practice  items  there  is  no  time  limit  imposed  on  the  subject.  During  the  actual 
test,  the  subject  must  respond  to  items  within  a  fixed  limit.  Failure  to  respond  in  time  results 
in  an  item  being  scored  as  incorrect.  The  interstimulus  interval  is  decreased  in  each 
successive  group  of  test  items,  thus  increasing  task  difficulty. 

The  scoring  procedure  for  this  test  produces  several  response  accuracy  scores  including 
the  number  of  correct  responses,  number  of  nonresponses,  number  of  delayed  correct 
responses,  and  number  of  incorrect  responses.  The  DTG  test  is  designed  to  measure  selective 
attention  and  response  orientation.  The  instructions,  practice  and  test  items  require  about  20 
min  to  complete. 
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